Field of Art
The present invention relates generally to increasing the operational efficiency and extending the estimated operational lifetime of flash memory devices, and in particular to doing so by adaptively tuning operating parameters of flash memory chips and employing other less extreme preventive measures in the interim, via an interface that avoids the need for direct access to the test modes of the chips.
Description of Related Art
Computer systems have increasingly come to rely upon solid-state drives (SSDs) to replace or at least supplement hard-disk drives (HDDs) as key storage system components. Despite their higher cost (which continues to drop) and lower storage capacity (which continues to increase), SSDs currently offer a variety of advantages over HDDs, including lower power requirements, faster access times and greater shock resistance, among others.
Most SSDs rely upon persistent (non-volatile) NAND-based flash memory, which employs floating-gate MOSFET (FGMOS) transistors to represent binary “0” and “1” values. Newer flash memory technologies (e.g., “MLC” or multi-level cell, and “TLC” or triple-level cell) enable storage of more than one bit per transistor cell. Alternative flash-based technologies, such as NOR-based flash memory, are also employed on occasion for specific applications, such as code execution, due to its faster read speeds.
In any event, a flash memory device, such as an SSD, typically includes a flash memory controller (or flash controller) and one or more flash memory chips. A “host” system (e.g., a laptop or desktop PC, or an enterprise server) may employ one or more internal and/or external SSDs for persistent storage. To access that storage, a host system sends “system memory commands” to an SSD, requesting that the contents of a particular file be read, written or “deleted” (i.e., “freeing up” the area of memory it occupied for future use by other files). SSDs implement those system memory commands via their flash controller, which associates a file with the particular flash memory chips and areas of flash memory in which that file is or will be stored, and ultimately sends many lower-level “controller memory commands” to one or more flash memory chips in the SSD.
It should be noted that certain characteristics of flash memory affect the nature of these controller memory commands. For example, NAND-based flash memory can effectively change a “1” value to a “0” value, but not vice-versa. Thus, while even a single byte or word of data could be read from flash memory, a write operation could not overwrite currently stored data, because the new data might require that a “0” bit be changed to a “1” bit. Instead, flash controllers employ what is known as a “program/erase cycle” or “P/E cycle,” which erases an entire “block” of memory (setting all bit values in that block to “1”), effectively “freeing up” that block of memory to be written in the future—i.e., by writing only the desired “0” values while leaving the “1” values unchanged.
But, performing one or more P/E cycles every time a host system issues a write command would be an inefficient use of time, leaving the host system waiting a relatively long time for the write to complete. Flash controllers therefore attempt to “free up” or “recycle” large numbers of blocks of flash memory as part of a “background” process (i.e., as part of their standard “garbage collection” process that reclaims unused areas of flash memory) so that a write command issued by a host system rarely directly necessitates a P/E cycle. The initial write operation to a “free” block is generally considered to be the “program” step (P) of a P/E cycle, while the subsequent “erase” (E) of that block may not occur until much later—e.g., during the flash controller's garbage collection process.
When a host system erases a file, the flash controller merely “marks” those portions of blocks containing the data as “unused.” But when a host system overwrites a file (e.g., when a user modifies and saves the file), the flash controller must actually “move” the contents of that file by writing it to free blocks of flash memory, causing the prior blocks to be marked as unused and later reclaimed as part of the background garbage collection process. While all blocks are initially “free” to be written, write commands issued by a host system over time will indirectly cause a flash controller to perform multiple P/E cycles on various blocks of flash memory (a phenomenon referred to as “write amplification”). Eventually, once “almost all” (e.g., 90%) blocks have been written once, the flash controller, during its background garbage collection process, will accumulate enough “valid” (as opposed to unused) data from multiple blocks to fill an entire block, “move” that data by writing the data into a free block, and then erase the multiple blocks, thereby creating additional free blocks ready to be written as a result of subsequent host system write operations.
Flash memory chips are typically organized into multiple “logical units” or LUNS, each of which physically includes multiple “blocks” of memory, where each block is effectively the smallest erasable unit of memory on the flash chip. Each block in turn is organized into multiple “pages” of bytes. For example, a 1 TB (terabyte) flash memory drive (e.g., an SSD or other flash memory storage device) might include 8 flash chips, each of which has a capacity of 128 megabytes (1 gigabit). Each 128 MB flash chip might be organized into 4 LUNs (32 MB each), where each LUN includes 128 blocks (256 KB each), each block includes 64 pages (4K each), and each page includes 8 sectors (512 bytes each).
An SSD's flash controller serves as an interface between the individual flash memory chips in the SSD and the host computer system. As will be discussed in greater detail below, P/E cycles take a toll on the flash memory, effectively reducing the “operational lifetime” of individual flash memory chips, and thus of the entire SSD (often measured “after the fact” as the total number of P/E cycles performed on the flash memory before it “wears out”). For example, blocks of flash memory “wear” or “degrade” (used interchangeably herein) each time they are written and become less capable of being written reliably over time (a problem known as “endurance”), and, depending upon their use (e.g., in a USB thumb drive), may not be able to retain data reliably for long periods of time between writes (a problem known as “retention”). In addition, read operations on a cell of flash memory often corrupt or “disturb” the contents of neighboring cells (a problem known as “read disturb”). As will also be discussed below, flash controllers employ various techniques to manage an SSD's flash memory chips and address these endurance, retention, read disturb and other problems, so as to improve operational efficiency and extend the operational lifetime of the SSD.
It should be noted that flash memory chips also include, in addition to the physical flash memory itself, a “state machine” that implements the controller memory commands received from the flash controller. Although one could implement this state machine as a microcontroller, it is important to distinguish the functionality performed by this state machine (i.e., interpreting controller memory commands—e.g., to read and write particular areas of flash memory, and erase particular blocks) from the higher-level and more extensive functionality performed by an SSD's flash controller—such as formatting the flash memory, performing garbage collection and addressing endurance, retention, read disturb and other problems, as well as implementing system memory commands by effectively “translating” them into many lower-level controller memory commands targeted to the particular areas of flash memory in which a file's contents are or will be stored.
In addition to its state machine, flash memory chips also include control registers to store “operating parameters” (where each particular LUN on the chip has an associated set of operating parameters) that are employed by the state machine to implement controller memory commands. In order to implement any controller memory command (e.g., read, write, or erase), the state machine applies an “electrical stimulus” to the flash memory itself. Such electrical stimuli include voltage levels, for example, which are applied for a particular period of time. These levels not only depend upon the type of operation being performed (e.g., read, write or erase), but may also vary over time (e.g., increasing voltage upon a retry after a failed read operation).
The state machine applies these varying levels of electrical stimulus in accordance with the values of the operating parameters (associated with the designated LUN) stored in the control registers of the flash memory chip. For example, typical operating parameters include threshold voltage levels that differ for read, write and erase operations. They also may include start and end voltage values for each operation, as well as a duration or incremental rate/amount of change in value over time, e.g., instructing the state machine to increase a particular voltage level gradually over time (but not in excess of threshold amounts). Voltages might range, for example, from 0 to 35 volts. Other operating parameters may include, for example, a maximum number of retries (e.g., 15) before a read operation is deemed to have failed, and pass voltage values for unselected word lines, among others. Essentially any constraint on the electrical stimulus applied by the state machine can be implemented in accordance with operating parameter values stored in the control registers of the flash memory chip.
It is important, however, to distinguish variations in electrical stimulus applied by the state machine from modifications to one or more of the operating parameters during the chip's operational lifetime. For example, a state machine typically employs a “retry” mechanism to verify write operations—by retrying the write operation until all memory cells have been verified or until a specified threshold number of retries is reached, in which case the write operation will be deemed to have failed. Over time, write attempts may occur more frequently, and more retries may be necessary to achieve successful verification. The number of retries may jump from 3 to 5 to 7 over some period of time. But, the operating parameter specifying the maximum number of retries (e.g., 15) may remain fixed. Similarly, one operating parameter may specify an incremental increase (e.g., 2 volts) in the voltage level for each retry, while another operating parameter specifies the maximum voltage for write retries (e.g., 20 volts). Even though these operating parameters are designed to remain fixed, the electrical stimulus applied by the state machine (e.g., the voltage level applied during a write) will vary over time during each subsequent retry—e.g., increasing from 5V to 7V to 9V, etc. —but not exceeding 20V.
As a general rule, the operating parameters of flash memory chips are determined (with respect to each LUN) when each batch of flash memory chips is manufactured, and they typically remain fixed thereafter during the operational lifetime of the flash memory chips. A flash memory chip's operational lifetime can be said to begin after it is manufactured and incorporated into a flash memory device, such as an SSD (though it will not likely “age” or degrade significantly until it is powered up and used in the field). As a practical matter, an SSD's operational lifetime can be said to be coextensive with (and equivalent to) the operational lifetime of the flash memory chips contained therein, as the SSD cannot typically function without all of its flash memory chips (unless overprovisioned).
When a batch of flash memory chips is manufactured, the manufacturer typically performs diagnostic tests on one or more “test chips” in an effort to determine an “optimal” set of operating parameters (with respect to each LUN) that meet desired specifications—e.g., reliable read and write operations for 10,000 P/E cycles with a retention time of 6 months. This is typically accomplished by placing each chip into a “test mode” to enable the issuance of diagnostic commands that access and modify the chip's operating parameters. Once this “offline characterization” process is completed and an optimal set of operating parameters is determined for each LUN, these same optimal operating parameters are stored in the control registers of each flash memory chip in that batch.
These operating parameters are typically not expected to be modified in the field, and test mode access is generally intended to be limited to use by the manufacturer and service technicians. However, if knowledge of the diagnostic commands and the mechanism for entering and exiting the test mode can be obtained (with or without the permission of the manufacturer), then modification of these operating parameters may be possible during the chip's operational lifetime. But flash manufacturers are not always willing to provide test mode access information to third parties, as it can reveal proprietary information such as the names, values and precise functionality of the chip's operating parameters.
Once flash memory chips are manufactured and incorporated into flash memory devices, such as an SSD, their usage in the field may vary widely during their operational lifetime. For example, a USB thumb drive may be written infrequently and powered down for long periods of time until its contents are read again (requiring relatively high retention). Enterprise SSDs, on the other hand, may be read and written frequently over time (requiring relatively high endurance).
As noted above, flash memory degrades over time, depending greatly on the nature of its use. What makes a set of operating parameters “optimal” for usage in a USB thumb drive may not be optimal for use in an enterprise SSD. Moreover, no single set of operating parameters (for a given LUN) is likely to be optimal during each “stage” of a flash memory chip's operational lifetime. As flash memory degrades, for example, higher voltages may be necessary to reliably write its contents. Conversely, lower voltages may be sufficient earlier in its operational lifetime, provided that they yield sufficient levels of retention. Finally, due to manufacturing variations, flash memory chips from the same batch (and even individual blocks of flash memory) may exhibit different levels of wear in the same application.
All of this suggests that no single set of operating parameters is likely to be optimal indefinitely, and that operating parameters therefore should be varied during a flash memory chip's operational lifetime. But, without an understanding of how flash memory degrades, it is difficult to determine how and when to modify the operating parameters within each LUN of a flash memory chip, and whether other less extreme preventive measures can be employed in the interim.
As noted above, “endurance” is one of the key problems exhibited by flash memory. The application of electrical stimulus (e.g., quantity and duration of voltage levels applied cumulatively over time) gradually degrades flash memory until it eventually “wears out” and can no longer be written reliably. In other words, flash memory degrades as a result of cumulative programming (P/E cycles) that apply varying degrees of electrical stimulus (referred to herein as “stress”) over time. Cumulative write and erase operations result in more frequent read errors and retries over time. Eventually, the number of retries may exceed a predetermined threshold number of retries.
The cumulative number of P/E cycles a flash memory chip (or component LUN or block of flash memory) has endured at any given time can be roughly analogized to the “age” of that chip. But the “cumulative wear” of a chip over time also depends upon the level of stress it endures during those P/E cycles. For example, higher voltages applied during a write operation will result in greater wear. One can thus estimate cumulative wear of flash memory over time (from P/E cycles) as a product of the cumulative number of P/E cycles and the level of stress applied to that flash memory.
As a general matter, the rate of wear (i.e., cumulative wear per P/E cycle) at any given time is proportional to both the number of P/E cycles and the amount of stress applied during those P/E cycles. However, this rate is not linear—due to variations in manufacturing and the fact that electrical stimulus varies over time based on the actual usage of a flash memory chip in accordance with its operating parameters. In short, no single factor can be said to determine or reflect the “health” of flash memory at any given time—i.e., its actual cumulative wear or how much life remains.
For example, two flash memory chips exposed to the same number of P/E cycles, but with different levels of stress over time, may exhibit very different levels of cumulative wear. In other words, they may be the same “age” but have very different levels of “health.” If their operational lifetime is measured (after the fact) as a cumulative number of P/E cycles before they “wear out,” then one may effectively live longer than the other (e.g., 50,000 P/E cycles as compared to 10,000 P/E cycles).
Moreover, variations in manufacturing may cause one flash memory chip to “wear out” faster than another, even though both were exposed to the same number of P/E cycles at the same stress levels (i.e., the same estimated cumulative wear). As will be discussed in greater detail below, certain “outlier” blocks of flash memory may wear at a faster rate than other similarly situated blocks—i.e., at a faster rate than expected based upon their estimated cumulative wear.
Although a cumulative number of P/E cycles is often used as a specification of a flash memory chip's estimated operational lifetime (just as a person's estimated lifetime may be said to be 70 years), this specification typically presumes fixed operating parameters and an assumed (typically “worst case”) usage scenario. Just as a person's “lifestyle” can affect his or her health, a chip's actual usage and exposure to electrical stimulus can affect its health. While a chip's “age” can be measured in cumulative P/E cycles, this factor alone is not necessarily the best indicator of the current “elapsed life” or health of that chip. Just as an old person may be more healthy than a younger person (taking into account various health factors, such as weight, heart rate, blood pressure, cholesterol and glucose levels, etc.), the health of a chip can be assessed more effectively by monitoring various indicators of wear or degradation.
Moreover, just as a person's health can improve (and lifetime be extended) by monitoring and treating these various health factors, so too can the health of a flash memory chip improve (and its operational lifetime be extended) by monitoring various indicators of wear over time (such as bit error rates, number of read retries and program and erase timing, as well as a current cumulative number of P/E cycles), and “treating” such wear by “prescribing” certain preventive measures to reduce the rate of wear, including identifying and resting outlier blocks of flash memory and varying the chip's operating parameters over time (as discussed in greater detail below).
In other words, while P/E cycles cause wear, they are far from the only indicator of the health of a flash memory chip (or of individual blocks of flash memory). Moreover, while the cumulative number of P/E cycles (“age”) always increases over time, the actual rate of wear may increase or decrease during any particular time period or “stage” of a chip's operational lifetime. In other words, while a flash memory chip's health may generally deteriorate proportionally with the cumulative number of elapsed P/E cycles, it may do so at a faster or slower rate (depending, for example, on when stress levels are increased and when certain preventive measures are employed).
It is thus apparent that the operational lifetime of a flash memory chip can be extended beyond the chip's specified estimated number of P/E cycles by adaptively varying operating parameters and taking other less extreme preventive measures in accordance with monitored indicators of health (and not just age) over time in an effort to slow the rate of (inevitable) wear.
Apart from the problem of endurance, another problem exhibited by flash memory is a limited retention time—i.e., the duration of time after flash memory is written until its contents can no longer be successfully read. This retention problem results from a leakage of charge that naturally occurs over time. Typical retention periods might range from 3 months to 10 years, depending upon the application.
But, the retention problem is somewhat inversely correlated with endurance. For example, the application of a higher voltage when writing flash memory results in a longer period of retention, but causes greater wear and thus a lower level of endurance. Moreover, frequent P/E cycles also limit endurance, but effectively minimize the retention problem because the relevant block of flash memory has relatively recently been rewritten.
Thus, one must balance the goals of maximizing endurance and retention when identifying “optimal” operating parameters and determining when to vary them, as well as when to employ less extreme preventive measures in the interim to slow the rate of wear. Prior attempts to address these problems have included “wear leveling” and “bad block management.” For example, wear leveling endeavors to distribute wear evenly to blocks of flash memory by tracking writes among blocks and moving data to different blocks in an effort to distribute wear more evenly. While this technique is generally effective in allocating wear evenly among blocks of flash memory and minimizing the number of “worn out” blocks, it does not reduce the overall level of wear, nor does it address the problems posed by outlier blocks. Bad block management (i.e., avoiding usage of blocks that are “dead on arrival” or wear out early) helps to address the problem of prematurely ending the operational lifetime of a flash memory chip—e.g., by reallocating the contents of “bad blocks” to unused portions of “good blocks” of flash memory. But it also fails to reduce the overall level of wear.
While others have attempted to increase the operational lifetime of flash memory chips by varying operating parameters over time (see, e.g., U.S. patent application Ser. Nos. 12/769,208 and 12/388,366), all of these approaches have relied solely on a cumulative number of P/E cycles to assess the level of wear of the flash memory (i.e., the “age” of the flash memory, as opposed to its “health”). Moreover, some have incorporated certain functionality into the flash memory chips themselves, in a manner that creates an inherent interoperability conflict with third-party flash controllers and flash storage devices.
What is needed is a system and a set of techniques that can assess a current level of health of a flash memory chip (or component LUNs, blocks, etc.) during its operational lifetime, and can modify its operating parameters accordingly (i.e., transition to a new “health stage”), as well as employ certain less extreme preventive measures in the interim, so as to increase the chip's operational efficiency and effectively extend its operational lifetime, and thus extend the operational lifetime of the SSD or other flash storage device into which it is integrated. One must also somehow procure test mode access to the chip, directly or indirectly, in order to ensure the ability to modify its operating parameters.